Update ipfs binary cache with upload support #10
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds a post bool that can be used to POST content instead of GET or PUT it.
need to use a multipart form to correctly insert the data. this mirrors how curl -Ffile=asdf works.
This makes a number of changes: - Use ipfs:// and ipns:// for URI, this is a proper URI and now documented by ipfs - Remove ipfs.hh, move everything into store - Store daemonUri, which is the, replacing individual API settings - Remove support for gateways - Remove nar info cache - ipfs should do all of the caching for us - Only retry getFile once - if it doesn’t work the ipfs daemon is probably down - Use /object/stat to check if file exists instead of cat
To check if IPFS daemon works, we need to do a test that should alway work. In the future, we may want to have a min IPFS daemon that we support, but for now any version will work.
This is okay for /ipns/ because we can always mutate it, but it’s a hard error for ipfs.
This is really slow, but is useful to test things. It does a few API calls to work: - /api/v0/add : to add the new file - /api/v0/object/patch/add-link : to insert the file into unixfs - /api/v0/name/publish : to publish the new ipfs object to ipns The last step is the slow part since we need to wait for it to clear.
unfortunately we don’t have a good way to handle this for now, avoid rethrowing and just accept 500
|
I am using |
this doesn’t wait for the final response, but updates ipns internally.
To avoid race conditions while doing /object/patch/add-link & /name/publish, we need to lock some state. This adds a bool for inProgressUpsert so that we can know when other threads are also trying to publish to IPNS.
|
added an ipfs-binary-cache-develop branch |
matthewbauer
changed the base branch from
old-ipfs_binary_cache
to
ipfs-binary-cache-develop
|
Setting offline=true in all ipfs calls speeds up IPNS a lot. I think that's probably the easiest way to deal with the slowness. I'm unsure how this effects the actual propagation of IPNS outside of localhost though. |
matthewbauer
changed the base branch from
ipfs-binary-cache-develop
to
ipfs-binary-cache-develop2
Ericson2314
changed the base branch from
ipfs-binary-cache-develop2
to
ipfs-binary-cache-develop
Ericson2314
reviewed
Jun 10, 2020
this is called at the end of a sequence of addToStores calls to tell the store that it can now commit the data.
This uses the Sync primitive to store an ipfsPath that is written to as we add stuff to IPNS. IPNS is still optional. When finished, we do a publish on the ipnsPath.
…o ipfs-binary-cache-with-upload
… into ipfs-binary-cache-with-upload-sync
It isn't very safe, so best to do it once right after the condition. If only we had real pattern matching...
…load-sync Update ipfs binary cache with upload support (sync version)
Ericson2314
approved these changes
Jun 11, 2020
Ericson2314
pushed a commit
that referenced
this pull request
Dec 23, 2020
This deadlocks ProgressBar, e.g.
# nix run --impure --no-substitute --store '/tmp/nix2?store=/foo' --expr 'derivation { builder = /nix/store/zi90rxslsm4mlr46l2xws1rm94g7pk8p-busybox-1.31.1-x86_64-unknown-linux-musl/bin/busybox; }'
leads to
Thread 1 (Thread 0x7ffff6126e80 (LWP 12250)):
#0 0x00007ffff7215d62 in __lll_lock_wait () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0
#1 0x00007ffff720e721 in pthread_mutex_lock () from /nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libpthread.so.0
#2 0x00007ffff7ad17fa in __gthread_mutex_lock (__mutex=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/x86_64-unknown-linux-gnu/bits/gthr-default.h:749
#3 std::mutex::lock (this=0x6c5448) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/std_mutex.h:100
#4 std::unique_lock<std::mutex>::lock (this=0x7fffffff09a8, this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:141
#5 std::unique_lock<std::mutex>::unique_lock (__m=..., this=0x7fffffff09a8) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/bits/unique_lock.h:71
#6 nix::Sync<nix::ProgressBar::State, std::mutex>::Lock::Lock (s=0x6c5448, this=0x7fffffff09a0) at src/libutil/sync.hh:45
#7 nix::Sync<nix::ProgressBar::State, std::mutex>::lock (this=0x6c5448) at src/libutil/sync.hh:85
#8 nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:131
#9 0x00007ffff7608cfd in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88
#10 nix::getCodeLines (errPos=...) at src/libutil/error.cc:66
#11 0x00007ffff76073f2 in nix::showErrorInfo (out=..., einfo=..., showTrace=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/optional:897
#12 0x00007ffff7ad19e7 in nix::ProgressBar::logEI (this=0x6c5440, ei=...) at src/libmain/progress-bar.cc:134
#13 0x00007ffff7ab9d10 in nix::Logger::logEI (ei=..., lvl=nix::lvlError, this=0x6c5440) at src/libutil/logging.hh:88
#14 nix::handleExceptions(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()>) (programName="/home/eelco/Dev/nix/outputs/out/bin/nix", fun=...) at src/libmain/shared.cc:328
#15 0x000000000046226b in main (argc=<optimized out>, argv=<optimized out>) at /nix/store/h31cy7jm6g7cfqbhc5pm4rf9c53i3qfb-gcc-9.3.0/include/c++/9.3.0/ext/new_allocator.h:80
Ericson2314
added a commit
that referenced
this pull request
May 9, 2022
reword section on Nix store
Ericson2314
added a commit
that referenced
this pull request
Apr 8, 2024
Fix and test deleting when lower store references
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
POC of how upload support can work